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The materials on the following pages were used to organize, create and review 
portfolio evaluations of 93 community-based, social action programs sponsored by 
National 4-'H, U.S. Department of Agriculture, and the Kellogg Foundation. The first 
document PROJECT PORTFOLIO DESIGN^ was used to direct local program leaders to 
assemble portfolios. The second document. PORTFOUO PAHEL REVIEW GUIDE, was 
used to direct 45 panelists in their review of the submitted portfolios five months later. 

Portfolio evaluation is becoming an increasingly popular means for evaluating 
individuals in educational settings (Herbert, 1992). However, using portfolios for 
pro gram evaluation is a newer challenge. Thus, the materials in this document are our 
initial approach to large scale program evaluation with this technique. Results of the 
use of these guides will be disseminated following our implementation 

Thc programs 

The Cooperative Extension System (a nationwide educational network of U.S. 
Department of Agriculture, state land-grant universities, and county governments) 
identified Youth at Risk as a priority for action (Extension Service-USDA, 1992). In fis- 
cal 1991 $7.5 million in federal funds allowed for creation of 70 programs in school-age 
child care and education reading and science literacy, and coalitions for high-risk 
youth (poverty, lack of parental and community support, and negative peer pressure). 
In 1992 an additional $10 million was used to maintain 68 established programs and 
create 25 new ones. The projects focus on prevention and intervention rather than 
treatment. While the programs vary considerably in content, they all provide activi- 
ties, education, care, and community involvement using extensive local partnerships of 
public and private agencies. 

The combination of 93 diverse, nationwide projects made an interesting evalua- 
tion problem- Each project included a cooperative evaluation design in its proposal. 
Each of these designs was judged to be adequate for accountability and feedback pur- 
poses. However, each plan was designed locally as a collaboration among community 
activists, public agencies, volunteers, and land-grant universities. Thus there was a 
range of designs from tight pre-, post- intervention measurement experiments to quali- 
tative descriptive. The challenge for overall evaluation across projects was to create a 
procedure for including the range of quantitative and qualitative designs, extract gen- 
eralizations about successful strategies for researchers and policy makers, and to pro- 
vide authoritative reassurance to local projects about the value of their work and, when 
necessary, formative feedback for improvement. 

Program cvaluatianL background 

Good program evaluation is a difficult task. Multiple audiences with different 
agendas are interested in results. Each audience needs to be spoken to with a slightly 
different language. Fortunately, participants and audiences for YAR projects have es- 
sentially the same goal: make a difference foi at risk youth Another evaluation diffi- 
culty is that results are promised in proposals, but circumstances change. Parts of pro- 
grams, evon new developments emerge as the most significant components in unex- 
pected ways. In addition to good plans and proposals, audiences for the YAR projects 
rely on good people, flexible operations, discoveries, and increased understanding of 
what is to be done. The value of projects is not limited to initial goal statements; evalu- 
ations need to reflect added value. Portfolios are a good way to increase the scope of 
evaluation to account for unintended outcomes, changes, complexit)^ and multiple au- 
diences. 



The need for increased scope in program evaluation comes about from recent 
critiques of evaluation systems of projects like the YAR initiatives. Ginsburg and col- 
leagues (1992) at the U.S. Department of Education criticized evaluations that 
"...showed a preoccupation with measuring overall program impacts, particularly test 
score changes. While achievement outcomes are important, they don't tell the whole 
story. 'Black box' evaluations that ignore program processes are particularly frustrat- 
ing...." (p. 24). Evaluations need to illuminate and document the operations of the pro- 
gram ("black box") such as who is involved, for what reasons and to what extent and 
what is the actual use of materials and events like. Portfolios give projects the chance 
to show the internal mechanisms that produce outcomes. Another criticism of 
Ginsburg, et al is that evaluations are weak when designed as single. large-scale, multi- 
issue studies of complex phenomena. "A single study incorporating all...issues is often 
unwieldy and inefficient compared with using different evaluations to collect intensive 
information on particular topics. Corroborative findings drawn from several different 
evaluations may also provide more credible evidence than results from a single study" 
(p. 24). 

Traditionally, evaluation designs emphasize an "experimentalist" approach. 
Certain outcomes are specified, conditions set for achievement, and standardized mea- 
sures of pre- and post-performance used While this approach is valuable, it is not ade- 
quate to all of the needs of evaluation activity (Levitan, 1992). "All efforts to quantify 
net impact have inherent limitations, because the estimates are subject to differing 
ranges of uncertainty and the applicability of estimates based on samples to a national 
program remains problematic" (Levitan. p. iv). The reality is that YAR projects are not 
tight experiments, but widely ranging local interventions. In addition, they depend on 
their local context for meaning and function. 

Still another criticism of large scale evaluation designs made by Ginsburg, et al. 
is that the findings of important studies too often are not well integrated into decision- 
making. The problem is that while much of value can be learned from multiple pro- 
jects, there is a need to i;.nk this information as directly as possible with persons who 
make decisions about future practice, funding, research and development 

Mctibods and techniques 

A portfolio based evaluation system was begun for the Youth at Risk projects by 
providing guidelines for portfolio construction (PROJECT PORTFOLIO DESIGN). 
Optional portfolios were assembled by local projects as extensions to their approved 
evaluation plans. Portfolios permit local projects to best tell the story of their project 
by including additional materials and evidence in a compact format Portfolios are a 
strategy to include tight experimental evaluation designs, but also to accommodate the 
actual diversity of projects, unplanned for learnings and changes of emphasis, and the 
limits of objective planning. While portfolios may not have the same compelling na- 
ture of tight causal cases for linking goals, actions and outcomes, they do siccommodate 
more of what actually happens in multiple project systems. The YAR portfolio evalua- 
tion fosters individual projects as multiple data sources: each program focuses on a 
slightly different set of the constellation of problems and solutions associated with the 
YAR program as a whole. 

Portfolios are only the first half of the innovative evaluation design. Review 
Panels provide the judgment necessary to use the valuable information contained in 
the portfolios (PORTFOUO PANEL REVIEW GUIDE). National Review Panels are made 
up of persons best able to recognize the value, merit and worth in the project portfolios. 
The 8-14 members for each Panel include national 4-H staff, project directors, Centers 
for Action directors, representative participants, and decision-makers such as public 
officials. Review Panels have several tasks. First is to acknowledge the valuable work 
done by the projects. Second is to recommend additions or changes to make projects 



more valuable. Third is to summarize patterns of success across projects in terms of 
ideas, activities, strategies, resources, and materials. Finally, Panels identify political 
themes for advocacy in public forums. 

For the first two functions. Panels will communicate their reactions directly to 
projects. These responses insure that project efforts are noted, portfolio contents are 
read, and both formative and siimmative evaluations are completed. Identification of 
successful themes, common problems, innovative strategies, successful programs, and 
participant needs will be given to educational researchers. The most immediate recipi- 
ents will be researchers collaborating with local projects. A second contact will be 
thi^ough academic papers and convention presentations. The third contact will be docu- 
mentation through the Educational Resources Information Center (ERIC). 
Identification of themes having political and social value will be summarized by the 
Panels, and given to agencies making public testimony as to the need for Youth at Risk 
type activities, additional and continuing funding, and emphases in public policy at the 
local, state and national levels that address the needs of the youth served by the pro- 
jects. 

Portfolio panel reviews use a valuable commodity: human judgment Real peo- 
ple in small interactive groups, having discretionary power, have been shown to re- 
sourcefully recognize value, compensate for change, make small judgments that enable 
larger questions of value to be answered, and make their decisions reliably and usefully 
(Peterson. 1988). 

Ginsburg, A., McLaughlin, M.. Plisko, V. and Takai, R. (1992). Reinvigorating pro^jram evaluation at 

the U.S. Department of Education. Educational Researcher, 21[3), 24-27. 
Herbert, E. A (1992). Portfolios invite reflection-from students ancfstaff. Educational Leadership, 
58-61. 

Levitan. S. (1992). Evaluation of federal social programs: An uncertain impact. Washington. DC: 

Center for Social Policy Studies, The George Washington University. 
Peterson, K.D. (1988). Reliability of panel judgments for promotion in a school teacher career 

ladder system. Journal of Research and Development in Education, 21 (4), 95-99. 
Peterson. K.D. (1990). DOSSIER: A computer expert system simulation of professional Judgments 

on schoolteacher promotion. Journal of Educational Research. S^3), 134-139. 
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I. What is a Youth at Risk project portfolio? 

a compressed* data collection that documents the 
activity, preparation, and payoffs of a complex 
project. 

--a portfolio is a optional tool to expand your project 
evaluation plan in the approved proposal. 

"the portfolio does not replace the evaluation plan, but 
allows your project to best tell your own story with 
locally selected supplementary materials. 

-a portfolio is a format for presenting your evaluation 
plan results. 

--portfolios should look a little to a lot different from 
each other, but fit your project. 

--*compressed means that not everything about a project 
is included (like in a scrapbook): but rather a care- 
fully selected sample of descriptions, documents, 
data, and statements are arranged to best represent 
your project. (see section V; Portfolio Limits). 

-portfolios can explain your project, give perspective 
to evaluation results, add activity information, 
highlight the most valuable project parts, explain 
changes, and show off examples and ideas. 

-while evaluation and portfolios are key parts of your 

project, they should not grow so much in importance 
that they stunt the work of your project with at risk 
youth. Your project is important for what it does and 
accomplishes. The portfolio is just a reflection of 
this activity-not the dominating project activity, 
(see section V- Portfolio Limits). 

-portfolios are only half the evaluation review process. 
People to read and use the information are the other 
half. (see sections VII and VIII on Review Panels). 
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II. What is the purpose of a portfolio? 
it will help to: 

-tell the story of your project 

-get feedback to optimize your work 

-monitor, provide accountability 

-show the important ideas in your project 

-build a national economic and political case 

for more work like yours 
-lead to recommendations for other workers 

to follow, add to knowledge of effective 

strategies 



III. What goes into a portfolio? 

project-selected materials that tell your story 
(contents are not uniform for all projects) 



logs 



state: 



needs 




ots) 



CS 



List of possible portfolio contents to consider: 



yi^hat combination will best 

activity logbook 
project statement 
photographs 
newspaper clipping 
participant satisfaction 
standardized tests 
questionaires 
work samples 
self-concept data 
client statements 
product checklists 
competitors 
case studies 
what to do differently 



tell your story? 

survey of satisfaction 
prose descriptions 
videotape 

statistics, graphs, charts 
needs analyses 
pre-, post-tests 
accounting for objectives 
attitude measures 
evidence of production 
testimonials 
cost data 
bibliographies 
staff, collaborators 
attitude change 



Limits to a portfolio 

a portfolio is not an unorganized scrapbook 

--complex projects have been well represented in 

less than 18 bound pages, with several pages of 
appendices. Remember, real people will need 
to read, understand, and use what is presented. 

- some projects may need to include a folder, because 
of materials that cannot be bound. 

--certainly a portfolio should be no more than a small 
box or carton, even if it contains reports, work 
samples, and videocassettes. 

--slickly produced reports do not make a portfolio 

impressive; good ideas, important work, support 
for at risk youth certainly do. Think about what 
is valuable-flash and polish or projects that 
enhance the lives and futures of youth? People 
(the Review Panels) can tell the difference! 
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VI. Using a portfolio to account for goals and objectives 



A. It's important for your project to aim for cfear, 
ambitious, and elevating goals 

summarize your project aims in easy- to -find statements 

B. Objectives provide unambiguous, measureable, and 
easily communicated outcomes as indicators, targets, 
and foci 

—organize (or write) statements about what successful 
outcomes for your project will enable the youth you serve 
to do 

C. It's important to document attainment of your goals 
for acknowledgment, emulation, and reassurance 

—directly and succinctly show how your goals, activities, 
and outcome measures link to each other 

D. Not all important outcomes are known before your 
project occurs 

—describe the significant but unplanned-for outcomes 
that make your project valuable and notable 

E. Not all your clear, , .ibitious and elevating goals will be 
attained 

--show the unattained goals and briefly explain what happened 

F. It is helpful to account for your original goals and 
objectives (to learn from your experience and 

to determine the overall value of your project) 

—some goals & objectives attained (what evidence?) 
—some partially attained 
—some altered (why?) 
—some discarded (why?) 
—some added (why?) 
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they go to national Review Panels 
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Project 
Portfolio 




feedb ack. 



acknowledgement 
of 

important work 



Review 



Panel 



Researchers 



Summary of 
promising 
ideas, 
practices 



Case for continued 
support & funding 
for project activities 
in the society: schools, 
government, & 
private agencies 



VIIL What are Review Panels ? 

Review Panels are 8-14 person groups of 4H staff 
Center and Project directors^ community and 
participant representatives, scholars and public 
representatives that look for value, merit and 
worth in project portfolios. 

The purpose of a Review Panel is to judge the value, 
merit, and impact of your project—and to acknowledge 
these findings to you. Additionally, the Panel 
summarizes the successes, promising ideas of all 
all projects so that these summaries can be sent to 
researchers and decision-makers in the society. 
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PORTFOLIO PANEL REVIEW GUIDE 



YouthH 
At Risk 
Initiative 



BuBding Strong Families, Competent KWs, and Caring Communities 



December 1-4, 1992 
National 4-H Headquarters 
Washington, D.C. 
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PURPOSES AND EXPECTATIOKS 

(mailed to participants and presentation: 30 minutes) 

The purpose of this panel review is to provide human judgment as a basis for determin- 
ing the merit, value and worth of the projects as presented in portfolios. Portfolios are collections 
of reports, information, plans, products, activities, and results. They are assembled by the pro- 
jects to "best tell their own story." In this panel review process you wQl be guided to complete the 
following tasks: 

1. Survey and learn about portfolios . Fii^st. you will be guided to look through 
several actual portfolios and to talk about them in small groups of panelists. This wiU 
give you a chance to see whafs in them, think through some dilemmas about portfolio 
review, and set a few standards for review. 

2. Review the portfolios . You are expected to follow certain instructions, re- 
spond to the views of the funders, and communicate your own personal judgment about 
the adequacy, value and impact of the programs as presented. You are an expert in the 
area of programs for at risk youth, and thus should take in to account not only the goal 
statements given you, but your knowledge of needs of at risk youth, sound project or- 
ganization and functioning your own knowledge of community dynamics, and realistic 
expectations for actual programs. Your task is to judge the quality of the programs, and 
not merely the appearance or quality of the portfolios. 

Your judgment especially is needed to determine appropriatcnesR Project de- 
signers made certain promises in their proposal goal statements and intended activities. 
They should be accountable for these objectives and intentions. However, in a hierar- 
chy of value what may be more important than plans is the actual impact of the pro- 
grams as they happen for at risk youth. Thus, a change of plans in the direction ol 
more effectiveness can be positive. An unintended outcome that receives more attention 
should be noted and rewarded. Increased learning as a result of experience after the 
project begins should be supported. Finally, an effective project should not be limited 
by a less than perfect capacity to write proposals. In all of these situations, your hu- 
man judgment of appropriate program direction should discern, describe and acknowl- 
edge appropriate project activity and results which maximizes impact for at risk youth- 

3. Recognize and acknowledge impact You are expected to be an appreciative 
audience for the' efforts of the projects. Your task is to recognize good work and to ac- 
knowledge it. The job is to give credit where it is due. Your expertise and experience 
give you the ability to see quality and to respond to it 

4. Report information for improvement If you see areas for projects to improve, 
in planning, activity or reporting, record this valuable insight Your reactions may be 
identify problems, show missed opportunities, or give suggestions for changes in action- 
Report these simply and directly. 

5. Share what vou learned from the portfolios. Following work on individual 
portfolios, you will meet in small groups to discuss major ideas from the portfolios as a 
group. These ideas will be forwarded to researchers and advocates. 

6. Your Questions? 



JUDGING THE VALUE OF PROJECTS^ 



Assumptions: projects are complex, not completely defined, changing and dynamic, 
contain some unclear but important goals, and vary by view point 

Cnicnts for these projects are youth at risk, coalition members, communities of i>.e 
youth, society in general, and workers in the projects. The emphasis in determining 
TOluc should be more client-weighted than mansigemcnt -weighted That is, your con^ 
cern should be more for how worthwhQe the project was for clients rather than for how 
well the project was planned, organized, run, monitored, and presented (These latter 
concerns have some importance in the overall judgment, just lower priority). There 
should be more care about value to the client, than mere merit in the technical design 
as seen by the eye of the expert. 

Comparisons with other projects (here and elsewhere) are very important in determin- 
ing value. What other projects are able to do and accomplish, and cost, can be more 
important for judging value and worth than abstract ideals, theories or principles about 
working with at risk youth 

Costs for programs are comprehensive and include such things as dollar expenditures, 
time, volunteer effort maintenance, activities d/spiaced because of what was done, and 
disruptions of ongoing efforts. 



Questions that you niay want to consider include: 

what difference for at risk youth did the project make? 

what needs were met? what priorities does this population have? 

what is remarkable about this project? 

what strategies and techniques did the project use? are these defensible? 

how did the project document their plans, work, results? 

how was money spent? accounted for? was money well spent? 

how were project goals and objectives accounted for? 

how does this project compare with other projects on the above dimensions? 
any improvements come to mind? what should the project be praised for? 
anything missing in the project, or report? 



VahiaUc programs: 

meet ethical and legal expectations, 

show clear payoffs for clients (have good outcomes), 

have payoffs that last 

meet the demonstrated needs (and priorities) of the target populations, 
exhibit practices that lo gically meet the population needs (good process), 
are cost effective , 

may produce unexpected positive side effects but lack unanticipated 

negative side effects, 
have strategies that general ize in their effectiveness to similar problems in 

other settings, 
look good in comparison with competitor programs, and 
evaluate their own evaluations. 



Yom- judgment: take into account the best objective data from the portfolios, compare 
projects on key value questions, apply the standards of state-of-the-art projects of this 
kind, consider ethical questions raised, and synthesize your value determinations in 
each case. 



summary of program evaluation ideas of Michael Scrlven 



FUNDER GOALS AND OBJECTIVES 



Significant need is defined in terms of risk factors or barriers to youth reaching their 
full potential. Need is si gnificant when two or more risk factors interact. Risk factors 
include, but are not limited to: poverty, substance abuse, teen pregnancy, illiteracy, 
school dropouts, AIDS, and homeless youth. 

Role of funded projects: 

A. Establish community based programming for youth and families at risk. 

B. Build community collaborations that define problems, find solutions and gain 
effectiveness in the community. Collaboration members should share owner- 
ship, work toward common goals, and contribute equally. Collaboration DOES 
NOT mean Extension runs the program and other agencies/organizations serve 
as an advisory group. 

C. Gain community attention for the need to actively address the needs of youth 
and families at risk. 

D. Demonstrate an interdisciplinary approach from across Extension program ar- 
eas, university academic units, and where applicable, between 1862 and 1890 in- 
stitutions. 

E. Serve as a model for successful youth development education programs, which 
requires, among other things, long term, intensive program involvement. 

F. Utilize a variety of strategies to make the project self-sustaining after Federal 
funding ends, and integrate into ongoing community and Extension program- 
ming. 

G Increase protective factors and reduce risk factors. 
Objectives: 

A. Institutionalize Youth At Risk programming to insure program continuation 
beyond Federal funding, and integration into the community and ongoing 
Cooperative Extension Service programming. 

B. Increase the employability of youth. 

C. Enlist the support of local communities, public agencies, the private sector and 
volunteers to fonn coalitions and partnerships that strengthen the ecological 
environment for youth {individual, family, peer, school, work, and community). 

D. Involve parents in educational programs which build parenting skills. 

E. Develop community-based programs with a major volunteer component that 
Involves youth in planning and Implementation. 

F. Increase the self confidence and decision making skills of youth to cope with 
adverse conditions and become mature contributing adults. 

G Reduce the long-term costs to society for medical care, welfare, and incarcera- 
tion by increasing the proportion of youth who transition into adulthood as 
productive, capable and contributing citizens. 
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POKTFOUO REVIEW 

Seventy-five portfolios were submitted for review of the 45 panelists. The 
panelists were assigned to three work teams. Portfolios were evenly distributed 
(exception: portfolios from one's ovm state were not included in any team). Each team 
had a work room, a facilitator, and informal meeting areas. Panelists were generally 
encouraged to review portfolios by themselves, make notes and then to gather in pairs 
or small groups to discuss their findings. 

Feedback forms were prepared as carbonless duplicates. Feedback was 
recommended to be 1-2 liners of noteworthy findings. Panelists were asked to pick two 
portfolios to "shepherd," a process of more intense review (1-2 hours) and presentation 
at the summary session on the last day of review. In addition, panelists were asked to 
review and comment on as many additional portfolios as practical (approximately 5-10 
others). 
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YAR PORTFOLIO FEEDBACK FORM 

(pUase wriu firmly <fe cUariy) 



Project Name, State and Contact: 



Focus: SACC Science/literacy Coalitions 



Year: 



What is important/valuable about this project? 



What suggestions/ideas do you have for this project? 



Focus Group Guide 
YAR Portfolio Bvaluation Study 

Purpose of focus group study: 

To extract and synthesize the wisdom of portfolio panel review members on programs 
for youth at risk. These people will have just completed three days of reading and dis- 
cussing 20-30 (each group) portfolio descriptions of $100K local, community-based, 
coalition projects for addressing the needs of youth at risk. Because of their occupa- 
tional expertise and their most recent experience of immersion in YAR program infor- 
mation, they will be in a position to furnish useful information and perspective for oth- 
ers interested in the topic. In particular, audiences for this information will be: advo- 
cates for YAR (project designers, lobbyists), researchers, and community activists. 

Size and number of groups: 

Total participants: 45; # groups: ^; Persons per group: 9 
Questions to be asked: 

What strategies, techniques are working to meet the needs of youth at risk? 
What questions would you be willing to collaborate with researchers in 
investigating? 

What issues, arguments should advocates emphasize? 
Description of session: 

The session should be informal, comfortable and interested in the views of the partici- 
pants. The sessions are a chance for participants to discuss their views, information 
and priorities in a social setting. The sessions have some elements of group interviews, 
but allow for the interaction of the participants. The emphasis should be more the 
spontaneous interaction of participants, and not over dominated by the moderator. Tne 
few questions are to focus the conversation; but the participants should do most of the 
structuring. Numbers of ideas, development of ideas are key. Moderators should note 
the relative importance, priority and group reactions to ideas— as well as the ideas 
themselves. 

Duration of groups: 90 minutes (slightly longer at option of moderator) 

Setting : Quiet room. Private (or, if large room, unaffected by other group), refresh- 
ments avaOable, newsprint & marking pens 

D ocumentation: moderator notes, audio tape 

Readings for moderators : 

Krueger, R.A (1988), Moderating skllla In Focus groups: A practical guide for applied researcfi. 
(pp. 72-90). Newbury Park, CA: Sage. 

Morgan, D.L. (1988), Conducting and analyzing focus groups. In Focus groups as qualitative 
research, (pp. 53-60). Newbury Park, CA: Sage. 

Stewart D. & Shamdasani, P. (1990). The focus group moderaton Conducting the focus group. In 
Focus groups: Theoiy and practice, (pp. 69-86; 87-101). Newbury Park. CA: Sage. 



